The present invention relates generally to data management systems and, more particularly, to platform-independent data structures for organizing, managing and accessing large quantities of data received from numerous non-homogeneous data sources.
Various data transfer schemes are known in the art. Two common examples are point-to-point data transfers and radial staging design. Characteristics of a point-to-point data transfer system include no common monitoring of data transfers, limited data reusability, advanced knowledge of both target and source data systems, fast implementation, limited data snapshot (i.e., point in time) capability, and various tracking and auditing mechanisms.
In a radial staging design system, a staging database is designed as a heap. Characteristics of a radial staging data transfer system include multiple data owners, limited standardization, limited snapshot capability, advanced knowledge of both target and source data systems, and advanced knowledge of staging database. The data staging process imports data as either streams or files, transforms the data, and stages the data for loading into data warehouses, data marts or operational data stores. The data staging process is driven by metadata, including business rules. Metadata is used along with administrative tools to guide data extractions, transformations, archiving and loading of data to target data warehouse and data mart schemas.
Such previous solutions cannot handle the complexity of the multiple data sources and volumes of data, especially when dealing with data from legacy applications. Such solutions require a good deal of manual support, constant re-design when business needs change, and an experienced staff to provide daily maintenance. There is a need for an invention that creates a data structure that is sufficiently flexible to eliminate the continual creation of new schema.